Joint mesh and texture compression using marginal analysis

In many computer graphic applications, a texture is mapped onto a meshed surface. Under complexity constraints, both texture and mesh need to be approximated. In this paper, we present a framework for the joint simplification of texture and mesh with respect to an error measure in screen space. In order to achieve optimal operating points, we choose two efficient simplification algorithms: Optimal multiresolution mesh simplification based on a quadtree structure and multiresolution view-dependent texture compression based on wavelets. Together, these two algorithms are used for joint simplification using an efficient heuristic based on marginal analysis. As an application example, we study the mapping of aerial orthophotographs onto a terrain model of the swiss alps, and show that the resolution of the terrain and the aerial photograph is efficiently traded-off. keywords: Compression Algorithms, Texture Mapping, Triangle Decimation, Wavelets, Marginal Analysis 1. PROBLEM FORMULATION In this paper we address a simple yet basic question of computer graphics: given a surface specified by a mesh and a texture to be mapped on that mesh, what is the resolution required for the mesh and the texture to achieve the best combined display quality? This question is relevant every time we are resource-constrained, and we need to simplify meshes and/or textures mapped on the meshes. The resource constraint might be computational (complexity of rendering meshes and mapping textures), storage space or bandwidth (bitrate needed to represent, transmit or update a complex model and its associated texture information) or both. In such cases, it is critical to choose the correct resolution or level of detail for both the mesh and the texture mapped on it. If more resource become available (e.g. in progressive downloads over the Internet), it is important to know if the mesh or the texture needs to be refined first. To make our discussion more concrete, consider a specific case where the interplay of texture and mesh resolution is particularly intuitive, namely very large terrain models with aerial photographs to be mapped on the terrain models. Consider two extreme cases. First, consider mapping a very fine texture (a very detailed aerial photograph) onto a very coarse terrain model. This gives an unsatisfactory result: While many details are available, the surface model is too simplistic compared to the texture resolution. At the other extreme, take a very detailed terrain model, but Figure 1: View-dependent compressed texture of Figure 3 mapped on simplified mesh seen from the viewer position. The field-of-view is depicted by the red box. with an overly coarse texture (e.g. one value only per surface element). The mesh is now very complex, but the texture is trivial, thus not reaching a good trade-off. The above two extreme cases hint at a better solution, where the correct trade-off between mesh and texture resolution is found. Let us now formalize the problem. Given is a surface model at full resolution M0, and a family of simplified models {Mi}i=1...N . We associate a cost function CM (Mj) to each model which reflects its complexity (number of bits needed to represent the model, or computational cost for rendering the model). Likewise, given a full resolution texture T0, and a family of simplified textures {Ti}i=1...M , we associate a cost function CT (Tj) to each texture, which measures its complexity (again, bitrate or computational cost of rendering). Finally, for a given pair (Mi, Tj), we define a distortion measure D(Mi, Tj) which evaluates the error in screen space between the image generated by the full resolution version (M0, T0) and the approximated version (Mi, Tj). Calling the image on the screen I(Mi, Tj) to reflect its dependence on the underlying model and texture, a possible distortion is the l2-norm, or D(Mi, Tj) = ‖I(M0, T0)− I(Mi, Tj)‖2 (1) Note that I(Mi, Tj) depends on the rendering process (e.g. lighting), but we assume the same rendering process for the various approximate images. The statement of the problem is now the following: Given a total budget for the combined complexity of the mesh and texture: